Fix saving large pipelines #1335
Conversation
|
🚀 Deployed on https://deploy-preview-1335--etna-docs.netlify.app |
Codecov Report
❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more. @@ Coverage Diff @@
## master #1335 +/- ##
==========================================
- Coverage 89.09% 88.95% -0.15%
==========================================
Files 204 193 -11
Lines 12636 12321 -315
==========================================
- Hits 11258 10960 -298
+ Misses 1378 1361 -17
... and 11 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
| @@ -230,7 +230,7 @@ def _save_metadata(self, archive: zipfile.ZipFile): | |||
| output_file.write(metadata_bytes) | |||
|
|
|||
| def _save_state(self, archive: zipfile.ZipFile): | |||
| with archive.open("object.pkl", "w") as output_file: | |||
| with archive.open("object.pkl", "w", force_zip64=True) as output_file: | |||
There was a problem hiding this comment.
Does it affects loading somehow? Do we have test for it?
There was a problem hiding this comment.
It doesn't affect loading, we have tests on saving/loading (not large files). Large files I tested manually.
As I understand the problem that with archive.open is that is should know about the size of the file to create a correct metainfo. We provide it with force_zip64 to make sure it creates metainfo for large file.
There was a problem hiding this comment.
Before submitting (must do checklist)
Proposed Changes
Add option
force_zip64during writing to archive.Closing issues
Closes #1318.